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Preface 



The Department of Homeland Security (DHS) Office of Inspector General (OIG) was 
established by the Homeland Security Act of 2002 (Public Law 107-296) by amendment 
to the Inspector General Act of 1978. This is one of a series of audit, inspection, and 
special reports prepared as part of our oversight responsibilities to promote economy, 
efficiency, and effectiveness within the department. 

This report addresses the effectiveness of controls to monitor the cost and schedule of 
Customs and Border Protection's Secure Border Initiative technology program. It is 
based on interviews with employees and officials of relevant agencies and institutions, 
direct observations, and a review of applicable documents. 

The recommendations herein have been developed to the best knowledge available to our 
office, and have been discussed in draft with those responsible for implementation. We 
trust this report will result in more effective, efficient, and economical operations. We 
express our appreciation to all of those who contributed to the preparation of this report. 




Richard L. Skinner 
Inspector General 
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Executive Summary 

We conducted an audit of U.S. Customs and Border Protection's 
control of its Secure Border Initiative technology program. Our 
objective was to determine whether the program office has 
implemented adequate controls to ensure that cost overruns are 
avoided and established milestones are met. To accomplish this 
objective, we reviewed four task orders for the program in fiscal 
year 2008, totaling approximately $267 million. 

Customs and Border Protection needs to improve its control of 
contractor activities on the Secure Border Initiative technology 
program. Specifically, program officials did not ensure that 
contractors maintain up-to-date information in the primary 
management tool designed to provide managers with advance 
information regarding potential cost overruns and program 
progress. In addition, SBInet program officials did not ensure that 
a program event was properly completed before progressing to the 
next event and did not adequately document their review and 
acceptance of accomplishments and criteria at program events. 
Finally, the low number of government personnel to oversee 
contractor activities increased the SBInet program office's risk that 
program cost and schedule could not be adequately managed. 
Consequently, the SBI program office's ability to ensure that both 
current and future program goals are accomplished is reduced. 

U.S. Customs and Border Protection has taken steps to improve 
Secure Border Initiative technology program oversight by using 
the Defense Contracting Management Agency personnel to assist 
with contract administration and reissuing important program 
documentation. During January of 2010 the Secretary directed the 
Department of Homeland Security to conduct a thorough 
evaluation of the SBInet program to consider options that may 
better meet border security needs. We are making four 
recommendations to Customs and Border Protection that will 
enhance the program office's ability to ensure that costs are 
contained, schedules are met, and performance requirements are 
accomplished. CBP concurred with the four recommendations and 
are in the process of implementing corrective actions. 
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Background 



In November 2005, the Department of Homeland Security (DHS) 
established the Secure Border Initiative (SBI). The SBI program is 
a comprehensive, department- wide effort to secure the Nation's 
borders. The U.S. Customs and Border Protection (CBP) was 
tasked with serving as agent for the execution of the SBI program. 

Since fiscal year 2005, Congress has appropriated more than $3.6 
billion for SBI. DHS estimates that the total cost to complete the 
acquisition phase of the program on the southwest border will be 
$7.6 billion for fiscal years 2007 through 201 1. Approximately 
$5.1 billion of the $7.6 billion is for the design, development, 
integration, and deployment of fences, roads, vehicle barriers, 
sensors, radar units, and command, control, and communications 
equipment. The remaining $2.5 billion is for integrated logistics 
and operations support. 

A component of the overall SBI effort is the Secure Border 
Initiative Net (SBlnet), a major acquisition program initiated to 
gain operational control of the borders by designing a new 
integrated system of technology, infrastructure, and personnel. In 
September 2006, the department awarded a three-year, indefinite 
delivery /indefinite quantity contract to the Boeing Company to 
integrate and implement a technology solution to monitor the 
southwest border. As of February 2010, CBP had awarded 13 task 
orders to Boeing, for approximately $1.2 billion for the SBI 
program. 

The SBlnet program office is responsible for planning, acquiring, 
and deploying the appropriate combination of technology and 
tactical infrastructure for border security. The program office is 
also responsible for ensuring effective oversight of the SBlnet 
program, including cost and schedule control, which entails 
analysis and reporting on program status information. Relevant 
and timely program cost and schedule data must be provided to 
and reviewed by program officials to detect early warning 
indicators of potential problems and to facilitate preventive or 
corrective actions. 

The department's ability to monitor SBlnet has been a continuing 
concern. In November 2006, we reported that the department 
needed to build the organizational capacity to manage the SBlnet 
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program and implement processes to mitigate workforce turnover. 
At program initiation, the department did not have the acquisition 
workforce required to plan, oversee, and execute SBInet. More 
recently, in June 2009 we reported that CBP had not established 
adequate controls and effective oversight of contract workers 
responsible for providing SBI program support services. 2 

In addition, in September 2008 the Government Accountability 
Office found significant risk of the SBInet program not meeting 
mission needs and performing as intended, as well as the risk of 
increased program cost and time-consuming system rework. 3 The 
Government Accountability Office attributed these risks to the 
continually changing scope and timing of SBInet capabilities, the 
absence of properly defined and managed requirements, the 
absence of a program schedule to guide the execution of the 
program and schedules that continually change, and ineffectively 
managed testing. During January of 2010 because of growing 
concerns regarding the efficacy of the implementation of SBInet 
planning and technologies, the Secretary of DHS requested a 
department-wide reassessment of the program. The objective was 
to identify alternatives that may more efficiently, effectively and 
economically meet border security needs. Subsequent to this 
reassessment the Secretary froze all SBInet funding beyond 
SBInet' s initial deployment to the Tuscson and Ajo regions until 
the rassessment is complete. 

Results of Audit 

CBP needs to improve its control of contractor activities on the 
SBI technology program. Specifically, program officials did not 
ensure that contractors maintain up-to-date information in the 
primary project management tool designed to provide managers 
with advance information regarding potential cost overruns and 
program progress. In addition, SBInet program officials did not 
ensure that a program event was properly completed before 
progressing to the next event and did not adequately document 
their review and acceptance of accomplishments and criteria at 
program events. Finally, the low number of government personnel 



1 Risk Management Advisory for the SBInet Program Initiation, DHS OIG-07-07, November 2006. 

2 Better Oversight Needed of Support Services Contractors in Secure Border Initiative Program,, 
DHS OIG-09-80, June 2009. 

3 Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering Key Technology 
Investment, Government Accountability Office (GAO-08-1086, September 10, 2008). 
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to oversee contractor activities increased the SBlnet program 
office's risk that program cost and schedule could not be 
adequately managed. Consequently, the SBI program office's 
ability to ensure that both current and future program goals are 
accomplished is reduced. 

Contractor Oversight Activities Need Improvement 

Project Management Tool Not Kept Up-to-Date 

CBP personnel did not ensure that current baseline information 
was entered into the Earned Value Management System (EVMS), 
the primary oversight system designed to provide management 
with advance information of potential cost overruns and schedule 
slippages. A baseline is time-phased information used to measure 
work performance. It provides the basis for all program-planning 
activities; cost estimates; and project and program status 
determination, analysis, and reporting. 

Office of Management and Budget Circular A- 1 1 , the Federal 
Acquisition Regulation, and DHS guidance require the use of an 
EVMS to monitor performance on major investments and systems 
under development, such as SBlnet. Also, the contract requires the 
contractor to provide the system that meets the criteria as defined 
in the current American National Standards Institute/Electronic 
Industries Alliance (ANSI/EIA) Standard 748-1998, Earned Value 
Management Systems, approved May 19, 1998. Together, these 
criteria require that task orders in support of programs that have 
assets in the development, modernization, or enhancement phase to 
use EVMS to measure the cost, schedule, and performance of those 
assets against the established baseline. Similarly, Guideline 8 of 
the ANSI/EIA Standard, as well as references within the 
contractor's Cost Management Plan and the Integrated Master Plan 
Manual, stipulate that work be baselined as soon as possible after 
the contractor receives the authority to proceed, regardless of 
whether an integrated baseline review has been conducted and 
unless contractual authorization says otherwise. An integrated 
baseline review is the process to establish and maintain an 
understanding between the contractor and the government of the 
baseline as a means of mitigating risk. 



4 Office of Management and Budget Circular A-ll, Preparing and Submitting Budget Estimates, August 
2009. 
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When implemented correctly, Earned Value Management (EVM) 
provides managers with the necessary information to ensure that 
day-to-day decisions keep program performance consistent with 
established objectives. Without accurate baseline information to 
reflect cost and schedule variances, management is hampered in its 
ability to make viable operational decisions. EVM assists with 
work planning, performing work activities according to the plan, 
and measuring accomplishments against the plan. EVM mitigates 
the risk of cost and schedule overruns, bringing visibility and 
advance warning of problems before tasks are completed, and 
providing opportunities for proactive corrective actions. EVM also 
provides a forecast of final program cost and schedule outcomes. 
Essential to EVM is a cost baseline — the time-phased budget 
information used to measure work performance. This cost baseline 
provides the reference point for all program-planning activities, 
cost estimates, and project and program status determinations, 
analysis, and reporting. 

We reviewed four fiscal year 2008 task orders each valued at over 
$20 million. For one of the task orders, the Integrated Logistics 
Support task order, the contractor was not required to use EVM 
because it was classified as a level-of-effort task order. The other 
three task orders showed that the EVM process had not been 
working as an effective management cost control tool for the 
SBlnet program. Outdated or incomplete baseline information for 
these task orders hampered accurate EVM tracking and subsequent 
cost control. Baseline information for the Arizona Deployment 
Task Order and incremental work in the System task order and 
Command, Control, Communications, and Intelligence Common 
Operating Picture task order were not updated in EVMS. The 
contractor and program office chose to wait until the integrated 
baseline review was conducted or incremental work was 
definitized before entering such data. Accordingly, managers were 
only able to review actual costs incurred with no basis for 
comparing actual money spent to projections, greatly increasing 
the risk that program managers could not identify cost overruns, 
measure progress, and track and schedule performance. 

Since June 2008, CBP has contracted with the Defense Contract 
Management Agency to provide surveillance of the contractor's 
EVMS. This surveillance was not in place for the FY 2008 
information on the task orders included in this review. Defense 
Contract Management Agency representatives are included in 
program meetings with the contractor and have served to promote 
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contractor cooperation regarding inclusion of baseline information 
in the EVMS by issuing Corrective Action Reports. 

Review and Acceptance Of Program Events Not Adequately 
Documented and Properly Completed 

SBInet program officials did not adequately document their review 
and acceptance of accomplishments and criteria at program events 
due to the absence of an established process for doing so. As a 
result, documentation does not exist to demonstrate that 
contractors produced deliverables, such as radar and camera units, 
that met project objectives and schedules. Also, SBInet officials 
did not ensure that a program event was properly completed before 
progressing to the next event increasing the risk of significant 
rework and associated project delays. 

Inadequate Documentation of Program Events 

Major system acquisitions like SBInet typically are divided into 
program events. These program events are identified in the 
Integrated Master Schedule and Integrated Master Plan and include 
target dates, milestones and tasks to be accomplished by all major 
System Program Office, System Prime Contractor, and other 
SBInet stakeholders. 

According to the SBInet Program Management Plan, management 
review of key milestones provides an effective mechanism for 
reporting, tracking, and managing project progress and ensuring 
that delivered products and services meet established project 
objectives and schedules. 

The SBInet program office does not have an established process 
for documenting and archiving historical information on 
government decisions and acceptance of program events. This 
reduces the program office's ability to ensure that a program event 
actually occurred, produced the intended results, and transpired 
within established timeframes, as well as whether deliverables met 
program objectives before the program progresses further. 

For example, the Preliminary Design Review milestone was closed 
as part of the entrance criteria to continue with the subsequent 
program event. There was no documented evidence of government 
acceptance and closure of this milestone prior to entering the next 
program event. (See Appendix C for a list of major program 
events.) 
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In another example, the program office was unable to provide 
documentation supporting the closeout of two of the five program 
events we requested for the Command, Control, Communication 
and Intelligence Common Operating Picture task order, and was 
not able to provide documentation reflecting the government' s 
review and resulting decisions for all five program events. The 
program office only maintained PowerPoint documents of the 
contractor' s briefings on program events. These documents do not 
provide adequate detail to facilitate proper contract management 
and oversight. 

Program Events not Properly Completed 

Program events have associated accomplishments and criteria that 
must be met to begin (entrance criteria) or successfully completed 
(exit criteria) before progressing to the next event in the program. 
Events are normally considered complete after all entry and exit 
criteria have been satisfied; all issues have been addressed and 
assessed, the status agreed upon; and an updated risk assessment 
has been completed. When program events are not successfully 
accomplished before progressing to the next event there is 
increased risk that significant resources may be wasted because 
identified deficiencies or other problems have not been properly 
resolved. Proceeding with events without being ready or without 
successful completion of previous events simply because the 
"scheduled date" occurs, is considered a "scheduled-driven" 
approach to project management. The Integrated Master Plan is 
event driven and not schedule driven and each program event 
should occur based on the completion of its supporting 
accomplishments and criteria supporting those accomplishments. 

SBInet program event criteria has not been properly satisfied 
before the program continued to move forward. For example, in 
early 2007, CBP accepted and closed out the Systems Requirement 
Review even though the event deliverables did not meet project 
objectives. This occurred because of the program office's desire to 
keep moving forward to meet established schedules, i.e. scheduled- 
driven. The Systems Requirement Review was one of the early 
major program events requiring the government and the contractor 
to develop, define, and agree on program requirements. In a 
December 2009 letter from the contractor to the SBInet Program 
Office, the contractor stated that premature acceptance and closure 
of the Systems Requirement Review event without successfully 
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completing its criteria, resulted in significant rework during both 
the detail design and test planning phases of SBInet program. 
More recently the Department's Acquisition Review Board took 
steps to ensure that the System Qualification Test program event 
was completed and all issues addressed before moving forward 
with subsequent program events. For instance, the SBInet Test and 
Evaluation Master Plan identified specific exit criteria to move 
from developmental testing to system deployment and testing 
under operational conditions. System Qualification Test is one of 
two key events in the exit criteria. System Qualification Testing 
disclosed five major deficiencies: 

1. Tower sway in windy conditions; 

2. Radar generated clutter; 

3. Radar circuit breakers frequently tripped; 

4. Camera image blurry in windy condition; and 

5. Frequent computer crashes. 

In September 2008, the DHS Acquisition Review Board issued a 
SBInet Acquisition Decision Memorandum mandating the 
completion of more system testing before proceeding to 
deployment. In February 2009, the Acquisition Review Board 
issued another Acquisition Decision Memorandum stating that full 
deployment at the first site in Tucson 1, and deployment at the 
second site Ajo-1, both in Arizona, were not authorized until the 
System Qualification Testing issues were resolved. Later, in May 
2009 the Acquisition Review Board approved deployment in the 
first and second sites after being briefed on the resolution or 
mitigation of the System Qualification Testing issues. However, as 
of February 2010 the SBInet Program Office has not yet 
considered the System Qualification Testing program event as 
closed and with all issues assessed although deployment of the 
technology solution to the first site in Tucson 1 commenced in 
May 2009. 

Moving forward with the SBInet program without adequate 
satisfaction of program events criteria and resolution of its 
significant issues could result in a deployed system that does not 
fully meet program objectives and the waste of significant 
resources. The absence of updated baselines and documented 
events review and acceptance dates makes it difficult for the 
government to hold contractors accountable for not meeting 
timeframes or not making progress towards accomplishing 
acquisition expectations. Also, improper closeout of program 
events increases the risk of project delays and wasted resources. 
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Assurance of due diligence for closure and documentation of 
government review decisions should be made for all SBInet 
program events to ensure significant resources are not wasted, and 
project objectives are met and accomplished within intended 
timeframes. 

Additional Staff Are Needed to Manage the SBInet 
Program 

Being fully staffed with qualified personnel is important for 
implementing effective controls over program costs and schedules. 
The low number of government personnel to oversee contractor 
activities increased the SBInet program office's risk that program 
cost and schedule could not be adequately managed. 

At the time of our review, the SBInet Program Office had only two 
schedule analysts and one earned value analyst on board 
performing Earned Value and Schedule Management activities for 
the entire program. These three employees were all Support 
Services Contractors responsible for important cost and schedule 
oversight activities, such as ensuring that the reported schedule and 
earned value information were accurate. 

For example, according to the SBInet Program Management Plan, 
the schedule management staff must develop and maintain a 
cohesive Integrated Master Schedule and Integrated Master Plan. 
Along with this, the earned value management staff is responsible 
for maintaining the Work Breakdown Structure, and for assisting 
with assessing compliance with the ANSI standards for EVM and 
performing earned value surveillance activities, which include 
developing, implementing, and maintaining program baselines. 
(See Appendix D for a list of key program documents and their 
purpose.) 

Since early stages of the SBInet acquisition, the SBInet Program 
Office has faced challenges in maintaining adequate staffing, as we 
highlighted in several prior audit reports. 5 In addition to these 
challenges, the SBI Program Officials stated that the initial 
assumption that commercial off the shelf technology would be 
available to cover SBInet needs, serving as a basis for determining 
staffing requirements, ultimately proved to be wrong. 
Consequently, staffing needs should be reassessed. 



5 OIG-09-80 and OIG-07-07. 

Controls Over SBInet Cost and Schedule Could Be Improved 

Page 9 



The low number of staff performing Earned Value and Schedule 
Management activities highly increases the risk of the program 
office not being able to properly control SBInet cost and schedule 
and timely react to operating issues encountered, such as the result 
of not using commercial off the shelf technology. During the time 
of our review, the SBInet acquisition was undergoing significant 
changes requiring updates to important program cost and schedule 
documents. The SBInet program office was operating without an 
approved Integrated Master Schedule, and the contractor was using 
a Work Breakdown Structure that had not been vetted by the 
government and was in non-compliance with ANSI Guideline 8 
requiring the use of baseline information as soon as possible. 

According to program office staff responsible for earned value and 
schedule management activities, the low number of personnel with 
adequate authority to perform earned value and schedule 
management activities in the program office reduced their ability 
to respond to such issues when they arose and affected their ability 
to complete all assigned tasks. 

To improve overall management and oversight of SBInet 
contractor activities, the SBInet Program Office was in the process 
of re- structuring in 2009. During the time of our review a 
government employee was added to serve as direct supervisor of 
the schedule and earned value analysts. Having a government 
employee as focal point for the analysts should improve the 
Program Office's authority to ensure contractor compliance with 
contract requirements, such as reporting accurate schedule and 
earned value information. In addition, the SBI Program Office was 
in the process of developing a long-range Human Capital Plan to 
describe the necessary competencies of the acquisition workforce, 
along with the individual skill sets and levels needed to execute 
and sustain current and future acquisition efforts. 

Recommendations 

We recommend that the Commissioner of Customs and Border 
Protection require the SBInet Program Office to: 

Recommendation #1 ; Ensure that no work effort, subject to 
earned value management system requirements, is performed 
without adequate corresponding performance measurement 
baselines in the Earned Value Management System, as required by 
prescribed guidelines. 
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Recommendation #2 ; Develop and implement a process to 
document government review and acceptance of program events' 
accomplishments and criteria. Documentation should clearly show 
program office evaluation and justification for approval and 
acceptance of all accomplishments and criteria for a program event 
to certify deliverables met project objectives and events were 
adequately completed before the program progresses. 

Recommendation #3 : Ensure that program events have been 
properly completed to include satisfaction of all entry and exit 
criteria; all issues have been addressed and assessed, the status 
agreed upon; and an updated risk assessment before proceeding 
with subsequent program events. 

Recommendation #4 ; Reevaluate the SBlnet program staffing 
plan and have an adequate number of earned value and schedule 
management analysts to support the current workload for the 
SBlnet acquisition and to implement all control duties assigned, as 
prescribed in the Program Management Plan. 

Management Comments and OIG Analysis 

CBP concurred with the four recommendations and the 
recommendations are considered resolved. The recommendations 
will remain open until CBP provides documentation of the 
measures taken to address them. CBP's response to the 
recommendations and our analysis is presented below. A copy of 
CBP's written response is included in Appendix B. 

CBP's Comment to Recommendation #1: CBP concurred with 
this recommendation but suggested revising the recommendation 
language to make it less broad. 

OIG Analysis: To further clarify the recommendation language 
we reworded it to "Ensure that no work effort, subject to earned 
value management system requirements, is performed without 
adequate corresponding performance measurement baselines in the 
Earned Value Management System, as required by prescribed 
guidelines." 

This recommendation is resolved but will remain open until CBP 
provides documentation of the measures taken to address this 
recommendation. 
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CBP's Comment to Recommendation #2: CBP concurred with 
this recommendation and reported measures that it is undertaking 
to address it. CBP set December 2010 as the due date for 
completion of reported measures. 

OIG Analysis: This recommendation is resolved but will remain 
open until CBP establishes that it has implemented the process and 
tools to manage program information requirements for program 
events. 

CBP's Comment to Recommendation #3: CBP concurred with 
this recommendation but stated that the report inaccurately 
concludes CBP did not assess all issues prior to commencing TUS- 
1 deployment. CBP also stated that not every single criterion 
needs to be fully resolved prior to advancing in the program and 
the Program Manager is to make a risk-based determination on 
whether, and how to advance based on the nature of open work 
(unfulfilled criteria) and the scope of the ensuing tasks. CBP 
stated a risk assessment was performed for the program resulting in 
the February 2009 Acquisition Decision Memorandum (ADM) 
mandating to commence only limited deployment at the TUS-1 
site. 

OIG Analysis: This recommendation is resolved but will remain 
open until CBP demonstrates that steps have been taken to 
preclude the recurrence of this issue. See OIG analysis to CBP's 
Comment #1 below, for additional OIG comments. 

CBP's Comment to Recommendation #4: CBP concurred the 
recommendation. CBP also commented that it expressed concerns 
regarding our assessment of the number of staff performing earned 
value and schedule management activities during the time of our 
review. They asserted that, at the time, they had an adequate 
number of analysts to perform all assigned control responsibilities. 

OIG Analysis: This recommendation is resolved but will remain 
open until CBP provides documentation defining roles and 
responsibilities of personnel dedicated to perform cost and 
schedule management activities and this information can be 
aligned with the organizational charts. 

Our report statement that additional staff were needed is supported 
by CBP's actions in March of 2009, when they reorganized the 
program office and initiated hiring actions to achieve an 
appropriate number of government earned value and schedule 
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management analysts to be deployed throughout the program 
office. 

CBP's General Comments to the Report: 

In addition to the comments to the recommendations, CBP made 
general comments to the report. The following paragraphs 
summarize CBP's comments and provide OIG comments. 

CBP Comment #1: CBP expressed concerns regarding our 
assessment that CBP proceeded with the deployment of the 
technology solution without having assessed all major issues 
identified at the System Qualification Testing program event. CBP 
also requested the OIG report be revised to (1) reflect the 
Department's ADM approval for limited TUS-1 construction 
documented in the February 2009 ADM, which they state the 
current report reflects as not authorized, and to (2) reflect that the 
limited deployment outcome was in fact based on an actual risk- 
based analyses and decision-making process regarding risk and 
remaining work as part of the formal acquisition review process. 

OIG Analysis: The OIG reported that CBP proceeded with the 
program when significant program events had not been properly 
completed. Their System Qualification Testing was a key program 
event that disclosed five major deficiencies. Based on our analysis 
of the data collected, as of February 2010 CBP had not yet 
assessed whether corrective actions for all the major issues 
identified at the System Qualification Testing program event were 
effective, yet they proceeded with deployment of the technology 
solution. 

Current language in the report was not intended to represent that, 
based on the February 2009 Acquisition Decision Memorandum, 
all deployment at the first site was not authorized. To further 
clarify, we reworded the report to reflect "full deployment at the 
first site in Tucson- 1 and deployment at the second site Ajo-1, both 
in Arizona, were not authorized until the Systems Qualification 
Testing issues were resolved." 

CBP Comment #2: CBP expressed concerns regarding our 
assessment of the number of staff performing earned value and 
schedule management activities. They stated we only talk about 
three analysts with database responsibility and forgot to 
acknowledge the "at least five contract analysts assigned to the 
Boeing task orders providing program control support throughout 
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2008." CBP also stated that we erroneously conclude a causal 
relationship between a perceived low number of analysts and 
increased program risk. 

OIG Analysis: Our finding that CBP had three analysts 
performing cost and schedule management duties and that this low 
number of personnel increases program risk, was based on 
documents provided by CBP and information obtained from 
interviews with program personnel and senior program officials. 

During our review senior program officials also stated that they 
would agree that the low number of program staff increases the 
risk to promptly respond to issues that arise in the acquisition 
process. 

CBP Comment #3: CBP commented that the report mistakenly 
states that "commercial-off-the-shelf (COTS) technology would be 
available to cover SBInet needs was a failed program assumption." 
CBP stated that the current SBInet solution in Tucson is 
predominantly COTS equipment. They also stated that the number 
of cost and schedule analysts needed in the program has no 
relationship to the COTS aspect of the program. 

OIG Analysis: Senior program officials stated during the course 
of our review that when originally determining the staffing needs, 
the assumption was made that "commercial-off-the shelf items 
would work and consequently large staff would not be needed. 
However, this assumption was later proven to not be valid." 

CBP's experience with Project 28 showed challenges using the 
commercial-off-the-Shelf technology equipment. Although, 
commercial-off-the-shelf equipment is being used, there have been 
major challenges with the integration of the commercial-off-the- 
shelf technology. System integration is part of the SBInet program 
needs. Consequently, we agree with the senior program official 
statement that the assumption that commercial-off-the-shelf 
technology would be available to cover SBInet needs was a failed 
program assumption. 
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Appendix A 

Purpose, Scope, and Methodology 

Our objective was to determine whether the SBlnet Program 
Office has proper control over the SBlnet program to ensure that 
schedules are met and costs are contained. We initiated this 
review in response to the FY 2007 Homeland Security 
Appropriations Conference Report, which directed the Inspector 
General to review and report on SBI contract actions in excess of 
$20 million. 

For our audit, we selected and reviewed the following four task 
orders associated with SBlnet contract actions over $20 million for 
Fiscal Year 2008: 

• Arizona Deployment Task Order, 

• Command Control Communications and Intelligence 
Common Operating Picture Task Order, 

• Integrated Logistics Support Task Order and, and 

• System Task Order. 

Together, the four task orders totaled approximately $267 million. 
Our office identified control processes and procedures in place and 
tested whether these controls were working as intended for the four 
task orders. 

We conducted fieldwork at Customs and Border Protection 
Headquarters in Washington, D.C. We interviewed program 
officials in the SBlnet Program Office and the SBI Acquisition 
Office, as well as government personnel from the Defense Contract 
Management Agency who currently provide certain contract 
administration services to DHS. We also reviewed key program 
documents and pertinent policies and procedures. 

We reviewed pertinent EVM criteria, obtained and reviewed fiscal 
year 2008 EVM Reports, and reviewed corrective action requests 
developed by the oversight agency, Defense Contract Management 
Agency. We met with schedule analysts, earned value analysts and 
project managers to understand how the program office establishes 
and uses milestones, schedules, and EVMS reports. We reviewed 
key program documents, such as the Acquisition Program 
Baseline, the Program Management Plan, the Systems Engineering 
Plan, the Integrated Master Schedule, and the Integrated Master 
Plan. Finally, we analyzed data to determine whether and how 
cost goals and milestones were met. 
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Purpose, Scope, and Methodology 



We conducted our audit from October 2008 through May 2009 
under the authority of the Inspector General Act of 1978, as 
amended, and in accordance with generally accepted government 
auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to 
provide a reasonable basis for our findings and conclusions based 
on our audit objectives. We believe that the evidence obtained 
provides a reasonable basis for our findings and conclusions based 
on our audit objectives. 
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pi U.S. Customs and 
1/ Border Protection 

April 29. 2010 



MEMORANDUM FOR RICHARD L. SKINNER 
INSPECTOR GENERAL 
DEPARTMENT OF HOMELAND SECURITY 



FROM: Assistant Commissioner 

Office oflnternal Affairs 
U.S. Customs and Border Protection 



^^rj^M^L * hip 



SUBJECT: Response to the Office of Inspector General's Draft Report 

Entitled, "Controls 0\er SBbftr Program Cost and Schedule 
Could be Improved" 



Thank you for providing us with a copy of your draft report entitled. "Controls 0\ er 
SBl/Jt'f Program Cost and Schedule Could be Improved." and the opportunity to comment 
on the issues in this report. The report identifies measures that the U.S. Customs and 
Border Protection (CBP) SBIw/ Program Office can take to enhance the overall 
effectiveness of cost and schedule controls. 



CBP acknowledges and appreciates the changes that were made by the Office of 
Inspector General tOIG) in its draft report based on the discussions that were held at the 
exit conference. 



The report contains four recommendations for CBP action. CBP concurs with the 
recommendations and believes the intent of Recommendations 1, 3 and 4 have been 
satisfied by actions taken to improve cost and schedule management at both the 
Department and the program level. 

As an initial matter, since the OlG's review in 2008. many significant accomplishments 
have resulted in substantive program management improvements and results, Chief 
among these accomplishments are: 

• Program Baseline: The Department of Homeland Security (DHS) Acquisition 
Review Board (ARB) baselined the SBIwf Block 1 program objectives, strategy, 
major supporting plans (e.g.. test and evaluation, integrated logistics support), and 
risks. Through a series of Acquisition Decision Memoranda (ADM), the 
Department approved Block I engineering, testing, and deployment activities 
based on thorough reviews of program accomplishments, remaining uncertainty 
and program risk, and the ultimate need to deliver effective capabilities to the 
Border Patrol. The primary result of these decisions was to stabilize the program 
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plans, and more importantly. the associated Boeing contract task orders where the 
vast majority of the funds arc spent- The lack of stability was the primary 
impediment to effective earned value management practices cited by the OIG. 

• Program Management: New leadership at the Secure Border Initiative (SBl) 
government and the Boeing contractor program management offices have 
significantly improved SBIhcj's focus, discipline, rigor, and overall effectiveness 
in planning and program control. As an example, the program leadership team 
implemented several comprehensive engineering and testing reviews as part of an 
"event-based" program plan, these added efforts included, among others, 
detailed Root Cause-Corrective Action analyses for major deficiencies, and 
increasing the amount and complexity of field system testing, in order to establish 
high confidence in system progress before continuing with significant future 
investments. Moreover, the SBl Program Office reorgam/.ed in early 200° to 
focus and solidify key system engineering and husiness management disciplines; 
SBl continues updating or creating program documentation to reinforce cost and 
schedule controls. 

• Program Progress: SBl and Boeing have completed - construction of the first 
Block 1 production sy stem in Tucson. Arizona: deployment of camera 
surveillance systems to Buffalo and Detroit under a fixed-price contract and 
nearly on schedule; establishment of a full-time maintenance and supply system 
for Mobile Surveillance Systems that has increased operational availability for 
these systems from an initial 55 percent rate to over 95 percent autilability today: 
and the Block 1 deployment to Tucson, built predominantly with commercially 
available technologies, is in operation today and the Border Patrol operators have 
experienced positive results with early operations to date. 

These accomplishments are evidence and outcomes of improved management controls 
established over the past year and a half. 

Not withstanding our concurrence with the report recommendations. CB1 1 takes exception 
to some of the findings and conclusions in the draft report, identified helow. 

• (page 8. I and 2 n,i whole paragraph) The OIG report inaccurately concludes that 
the SBl Program Office did not assess risks prior to commencing TLJS-1 
deployment. In fact. SB! reviewed risks and plans with the DHS ARB. and the 
Department rendered a risk-based decision to continue with a limited deployment 
activity while completing ongoing systems engineering and testing efforts. 
Specifically, SBl presented to the DHS ARB the initial System Qualification 
testing (SQT) results, including five major "deficiencies." Additionally, SBl 
presented clearly the system performance information gained, the information yet 
needed, and the associated tasks that could be initiated while SBl continued to 
address the remaining technical concerns and planning updates. The Department 
largely agreed with SBI's recommendations, and approved in January 2009 
(documented in the February 2009 ADM) the initial construction activities for 
TUS-1 sensor and communications towers and the command center upgrades. 
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Also, the Department deterred deployment of sensor pa> loads to TUS-1 and 
deferred all construction at AJO-1 until SBI completed and presented additional 
enginecrinj2 and lasting information. These departmental decisions reflect the 
actual risk-based analyses and decisionmaking that the Old cites as missing. 
Therefore, the Old draft report should be corrected to ( I ) reflect the Department's 
ADM approval for limited TUS-1 construction (current draft says TUS-1 
construction was not authorized), and (2) reflect that the limited deployment 
outcome was in fact based on an actual risk-based analyses and decision-making 
process regarding risk and remaining work as part of the formal acquisition 
review process. 

• (page 9, 2" J and 5 lfl paragraphs) The CHG incorrectly cites the full number of cost 
and schedule analysts deployed in the SBIw/ program, and erroneously concludes 
a causal relationship between a perceived low number of analysts and increased 
program risks. The 010 fails to acknowledge the full number of cost and 
schedule analysts deployed across the programs and projects. Throughout 2008. 
SBI had at least five contract analysts assigned to the Boeing task orders 
providing program control support, as well as three full-time support analysts 
charged with maintaining databases, analytical tools, and recurring status 
reporting. In formulating its conclusion regarding "low number of staff 

performing earned value and schedule management activities " the OIG only 

discusses the latter three analysts with database responsibilities. Thus, the finding 
is not accurate because there were actually up to eight analysts assigned for these 
responsibilities, and SBI continues to assert that this was an adequate number of 
analysts to perform all assigned control responsibilities. 

• (page 9. 4'" paragraph) The OIG asserts that SBI needs more cost and schedule 
analysts because of a failed program assumption that "commercial-off-the-shelf 
(COTS) technology would be available to cover SBIne/ needs." First, and 
contrary to the OlG's assertion, the current SBlmv solution in operation today in 
Tucson is in fact predominantly COTS equipment. So the SBI acquisition 
strategy relying on COTS remains valid. Additionally, the COTS aspect of the 
SBIwf program does not relate at all to the number of cost and schedule analysts 
needed in the program. so CBP recommends this paragraph he removed. 

A summary of CBP actions and corrective plans to address the four recommendations is 
provided below: 

Recommendation #1: Ensure that no work is performed without adequate 
corresponding performance measurement baselines in the Earned Value 
Management System. 

CBP Response: CBP concurs with the recommendation, l or this 2008 review, SBI 
shared two significant issues with the OIG that contributed to this deficiency: program 
volatility and breakdowns in following program processes. Over the past 18 months. SBI 
has made considerable progress in stabilizing the program and project plans following the 
Department baselining of the Block I program and associated Boeing contract task 
orders. This included joint Govemmeni-Boeing ell'ons diat established effective 
performance baselines for authorized contracted work, conducted refresher training 
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regarding earned value and baseline management practices, and ultimately reconciled 
procedures for maintaining performance measurement baselines consistent with program 
and contractor [Lamed Value Management System (EVMS) guidelines. 

CBP requests that the OlG consider updating the Recommendation 1 to read. " Ensure 
that all authorized work effort, subject to earned value management system requirements, 
is incorporated into the performance measurement baseline in accordance with prescribed 
system guidelines/" This change avoids prescribing EVMS policy for work that is not 
appropriately subject to EVMS (e.g.. low dollar value efforts on support contracts, level 
of effort tasking, firm-fixed price tasks, etc.) - the assertion that no work is to be 
performed without an EVMS baseline is simply too broad- 
Due Date: CBP believes the intent of this recommendation has been satisfied and 
requests closure of the recommendation. 

Recommendation 2: Develop and implement a process to document government review 
and acceptance of program events" accomplishments and criteria. Documentation should 
clearly show program office evaluation and justification for approval and acceptance of 
all accomplishments and criteria for a program event to certify deliverables meet project 
objectives and events are adequately completed before the program progresses. 

CBP Response: CBP concurs with the recommendLilion. I be SB! Program Office has 
made considerable progress in establishing formal processes and tools to manage 
program information requirements for key program events. First. SBl redoubled efforts 
with the Block I Program Integrated Management Plan (IMP) to clearly identify major 
program events, the required significant accomplishments, and the associated success 
criteria for each of the events. In addition. SBI recently stood up the Systems 
Engineering Division and is developing a Technical Review Manual (TRM) that 
documents the artifacts required for review and-'or fulfillment of significant technical 
reviews and program milestones. The review requirements are being integrated into the 
Program IMP to serve as the success criteria for the major program events, consistent 
with the OlG recommendation. A current organizational chart with functional 
responsibilities for each SBI directorate has been provided to the OIG electronically. 

Due Date: The estimated completion date is December 2010. 

Recommendation 3: Ensure that program events have been properly completed to 
satisfy nil entry and exit criteria, address and assess all issues, agree on the status, and 
complete an updated risk assessment before proceeding with subsequent program events. 

CBP Response: CBP concurs with the recommendation. Notwithstanding our general 
concurrence with this recommendation. CBP is concerned with the accuracy and context 
for the findings and analysis supporting this recommendation. In general, exit criteria for 
key program events are important to ensure thorough task planning and to assess progress 
and risks with continuing beyond the respective key events. Exit criteria, however, are 
not absolute, meaning not every single criterion needs to be fully resolved prior to 
advancing in the program. Rather, the Program Manager will make a risk-based 
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dele nn i nation on whether, and how. to advance based on the nature of open work 
(unfulfilled criteria) and the scope of the ensuing tasks. 

Such was the case when SHI and the 1)1 is \RB re\ie\\ed ihc initial N;. Man Qualification 
Testing (SQT) results and the live major "deficiencies" (refer to page 8 of the draft OIG 
report). For these specific reviews with the Department. SBI presented clearly [he 
information gained, the information yet needed, and the associated tasks that could be 
initiated while SBI continued to address the remaining technical concerns and planning 
updates. The Department largely agreed with SBI's recommendations, approved in 
January 2009 (documented in the February 2009 ADM) the initial construction activities 
for TUS-I sensor and communications towers and the command center upgrades (Note: 
the OIG draft report does not accurately report this guidance from the February 2009 
ADM). Because of open requirements (i.e.. higher risk), the Department deferred the 
deployment of sensor pay loads to TUS-I and deferred all construction at AJO-1 until SBI 
completed and presented additional engineering and testing information. 

The GIG report, therefore, inaccurately concludes (last sentence, first complete 
paragraph, page 8) that the SBI Program Office did not assess all issues prior to 
commencing TUS-1 deployment. SBI and DHS senior staff did assess risks, and 
rendered a risk-based decision to continue with a limited deployment acthity as 
articulated in the Departmental ADMs. 

Due Date: CBP believes the intent of this recommendation has been satisfied and 
requests closure of the recommendation. 

Recommendation 4: Reevaluate the SBI/it/ program staffing plan and ensure that an 
adequate number of earned value and schedule management analysts are in place to 
support the current workload for the SBlw/ acquisition and to implement all control 
duties assigned, as prescribed in the Program Management Plan. 

CBP Response: CBP concurs w ith the recommendation. In March 2009. the SBI 
Program Office reorganized, updated staffing plans, and initiated hiring actions for an 
appropriate number of government earned value and schedule management analysts 
deployed throughout the program office. In October 2009. the OIG was provided a copy 
of the then current SBI organization chart and a staffing roster of SBI personnel with 
position alignments. SBI considered the number of earned value and schedule 
management analysts sufficient to support the current workload, as prescribed in the 
Program Management Plan. 

Due Date: CBP believes the intent of this recommendation has been satisfied and 
requests closure of the recommendation. 
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With regard to the classification of the draft report. CBP has not ideniified any 
information within thi.s report that would warrant a "Kor Official Use Only" 
classification. Technical comments to this report were provided to the OIG 
electronically. 

If you have any questions regarding this response, please contact me or have a member of 
your staff contact Ms. Patty Qtimtana. CBP Audit Liaison, at (202) 344-1038. 
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Maior Program Events 


Scheduled 


Date 


Date of Review 




Date of 


Review 


AcceDtance bv 

i AVVVuvMIlVV J 




Review 


Took Place 


Government 


System Requirements 


1/19/07 


1/22-23/07 


Government close out 


Review 






letter is dated 3/20/07. This 








milestone was accepted 








with incomplete 








information. ^ 










Svstpm Preliminary Dpsipri 


4/27/2007 


4/27/2007 


This milpstonp was 


R PV1PW 






sn nnnspH to hp pIoqpH nut 
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comments were closed out 








but no Preliminary Design 








Review close out letter 








exists. This Preliminary 








Design Review closed as 








entrance criteria for 








Critical Design Review 










System Critical Design 


6/25/07 


6/3-5/08 


Government close out 


Review (BLOCK 1) 






letter is dated 10/31/08 










System Qualification 






Projected to begin in 


Testing 






9/2009. 



6 A close out letter is sent by the government to the contractor once the contractor presents documents 
addressing any deficiencies, comments, or changes that must be made for the government to officially 
accept the review. This is a not a formal process for documenting when the government accepts successful 
completion of a review, nor does the letter show what criteria was met to accept the review. 
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Key Program 
Document 


Document Description 


Purpose of Document 


Integrated 
Master Plan 


The Integrated Master Plan is an event-driven plan 
that documents the significant accomplishments 
necessary to complete project work and ties each 
accomplishment to a key program event. 


The Integrated Master Plan and its supporting detail 
schedule provides an overarching framework against 
which all work is accomplished. It documents all the 
tasks required to deliver a high quality product and 
facilitate success throughout the product's life cycle. 


Integrated 

Master 

Schedule A 

monthly 

deliverable 

required by the 

contractor. 


The Integrated Master Schedule is an event-based 
schedule and will include all major program, 
System Prime (contractor), and other stakeholders' 
schedule tasks, milestones, and dependencies. The 
Integrated Master Schedule will provide for 
automated import and export of component 
schedules and enable network and "what if" 
analysis to provide program management with 
timely visibility into key dates and milestones to 
ensure effective program management. The 
Integrated Master Plan and Integrated Master 
Schedule are fundamental management tools that 
are critical to performing effective planning, 
scheduling, and execution of work efforts. 


The key milestones in the Integrated Master Schedule 
represent significant events in the program and will be 
identified during the development of project-level or 
program office schedules. Program management 
review of key milestones ensures an effective 
mechanism for reporting, tracking, and managing 
project progress and ensures that delivered products 
and services meet established and agreed to project 
objectives. The Integrated Master Schedule is used by 
the Government and contractor team as the day-to-day 
tools for planning, executing, and tracking program 
technical, schedule, and cost status, including risk 
mitigation efforts. 


Work 

Breakdown 

Structure 


The Work Breakdown Structure is an exhaustive, 
product-oriented, hierarchical tree structure of 
tasks or deliverables that need to be performed in 
order for an investment program or project to be 
completed. It is the basis for structuring earned 
value management. The Work Breakdown 
Structure is an essential part of earned value 
management cost, schedule, and technical 
monitoring because it provides a consistent 
framework against which to measure progress. 


Investment programs, projects, and contracts, will use 
EVM against established Work Breakdown Structures 
at sufficient levels to assess performance against 
milestones and allocated budgets. The Work 
Breakdown Structure should be used to develop the 
cost estimate and the program schedule and to set up 
the earned value management performance 
measurement baseline. The Work Breakdown 
Structure can help define high-level milestones and 
cost driver relationships. The Work Breakdown 
Structure enables leadership to make better decisions 
about where to apply contingency reserve and where 
systemic problems are occurring. 
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Performance The performance measurement baseline represents 
Measurement the cumulative value of the planned work over 
Baselines time. The performance measurement baseline is 

essentially the resource consumption plan for the 
program and forms the time -phased baseline 
against which performance is measured. 



Baselines are necessary for defining the time -phased 
budget plan from which actual program performance is 
measured. Baselines are used to detect deviations from 
the budget plan and give insight into problems and 
potential impacts. Deviations from the baseline 
identify areas where management should focus 
attention. 
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ADDITIONAL INFORMATION AND COPIES 

To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4100, 
fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig. 



OIG HOTLINE 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal 
misconduct relative to department programs or operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, 
Attention: Office of Investigations - Hotline, 
245 Murray Drive, SW, Building 410, 
Washington, DC 20528. 



The OIG seeks to protect the identity of each writer and caller. 



